Unmanned driving vehicles driving in rainy environment face the following problems： the images collected by the car on-board camera contain rain streak noise， which reduces the target detection accuracy and difficulty in identifying key targets of the unmanned driving system. In order to solve these problems， a residual attention deraining network based on convolutional long short-term memory was proposed. Firstly， the Convolutional Long Short-Term Memory （CLSTM） units were proposed to learn the distribution of different scales of rain streaks. Then， the residual channel attention mechanism was used to extract the rain streaks. Finally， the extracted rain streak information was subtracted from the rain image to obtain the restored background image. To determine the optimal network structure， the ablation experiments of each network module were carried out， and the structure with best rain removal effect was selected as the deraining network. Through the continuous optimization of network parameters， the proposed algorithm was tested on Rain100H， Rain100L and Real100 datasets， the results illustrate that the Peak Signal-to-Noise Ratio （PSNR） of the proposed algorithm reaches 29.1 dB， 33.1 dB and 32.4 dB respectively， and the Structural SIMilarity （SSIM） of the algorithm reaches 0.89， 0.94 and 0.93 respectively. Experimental results show that through the additional supervision of the Generative Adversarial Network （GAN） discriminator， the proposed algorithm achieves an visible rain streak removal effect and enhances the environmental perception ability of unmanned driving system under complex rainfall condition.